The approaches described in this section are approaches that could be pursues, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion of this section.
Traditional software testing is constrained in its efficacy, maintainability, and modularity by the coupling of the test traversal steps with the expected behavior and checks performed on the system under test. For example, the test cases may be hard-coded to perform a select number of handcrafted scenarios which, if performed successfully, would indicate that the system is working properly. However, for any sufficiently large software project, even the best attempt at crafting varied and thorough test cases will fail to catch every edge case to which the software will eventually be subjected. Thus, traditional testing misses cases which the testers did not anticipate as potentially causing the system to exhibit unexpected behavior or did not think to check. Consequently, software is often released to the public with programming errors which should have been caught during the quality assurance stage of software development.
Due to the foregoing, there is a need in the industry for a randomized testing methodology that is randomized over either or both flow and data. Because randomized tests cannot be expressed or represented before execution as a series of actions and data and their associated expected outcomes (because actions and data are not pre-determined), a randomized test flow necessitates a software testing methodology that can separate test steps (or stimulus) from the checks performed on the system under test. By separating test stimulus from checks, stimulus may be selected randomly during the test progression and checks may be invoked and performed in response to the steps selected. In this way, since a randomized test generated by a system with separated stimulus and emulation/checking tests an arbitrarily lengthy, complex, and interesting set of scenarios generated on the fly, the test bypasses the limitations of premeditated case generation, and checks every possible element of the system at each step of the test. As test writers will never premeditate all interesting cases, and thousands of checks can never practically be programmed into every test by hand, efficacy and power of software testing can be significantly increased.